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(57)Abstract: 

PROBLEM TO BE SOLVED: To provide a video monitored 
during note generation or an important part of other data 
stream with a simple and accurate retrieval mechanism. 
SOLUTION: A recorded video is accessed from a printed 
note or summary introduced from the video. Analyzing the 
recorded video automatically generates a summary and a 
user for a device generating a note in digital ink and video 
provides a remark. The note and the summary are printed 
together with a data glyph providing a time-based index or 
offset to the recorded video. The index or offset is retrieved 
by scanning the glyph of printout. Glyph information can be 
imbedded in the printout by many methods. The accessed 
video is reproduced by a summary interface of a note 
generating device or a device of a web browser type. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 

[Claim(s)] 

[Claim l]An interface which accesses a data stream, including an output, comprising: 
At least one object corresponding to at least a part of data stream. 

A code relevant to each object which specifies said a part of data stream corresponding to said object. 

[Claim 2]The interface according to claim 1 embedded at each object to which said code corresponds. 

[Claim 3]The interface according to claim 1 for which said output includes further a table which makes 

each of said code correlate with offset of said data stream. 

[Claim 4]The interface according to claim 1 in which said output is print-out. 

[Claim 5]The interface according to claim 1 in which said data stream is video. 

[Claim 6]The interface according to claim 1 in which said code is data glyph. 

[Claim 7]The interface according to claim 3 in which said table is data glyph. 

[Claim 8] The interface according to claim 1 with which each of said code is arranged at a position 

corresponding to a code to which said data glyph carpet corresponds including a data glyph carpet as for 

said object. 

[Claim 9]The interface according to claim 8 with which said data glyph carpet is arranged in relation to 
said time line so that a code of said data glyph carpet may be equivalent to a point on time line also 
showing suitable offset of said data stream. 

[Claim 10] A formation method of an index for access to a data stream characterized by comprising the 
following. 

A step which specifies at least one object relevant to a data stream. 

A step which determines offset of said data stream corresponding to each related object, and a step 
which forms an index which specifies each offset and a corresponding related object. 

[Claim 1 1]A method according to claim 10 of containing further a step which outputs each of said 

specified object relevant to a code which specifies said corresponding offset. 

[Claim 12]A way according to claim 10 said data stream is video or multimedia data files. 

[Claim 13] A way according to claim 10 said object is at least one of user notes attached with reference 

to a key-frame of said data stream, a frame which said data stream chose, and said data stream. 

[Claim 14]A method according to claim 13 of containing further a step which considers a set of a related 

digital ink stroke as user notes, and carries out grouping. 

[Claim 15] A method accordmg to claim 13 by which each arbitrary digital ink strokes of one group are 

formed for said step which carries out grouping of a user within a predetermined time interval including 

grouping to a set of a digital ink stroke further. 

[Claim 16]A data stream accessing method comprising: 

A step which scans an object corresponding to a data stream. 

A step which determines offset of said data stream based on said scanned object, and a step which 
accesses said a part of data stream corresponding to said determined offset. 
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[Claim 17] A method comprising according to claim 16: 

A step which reads a code relevant to said object which said step to determine scanned. 

A step which accesses this offset from a table which makes said related code correlate with offset. 

[Claim 18] A method comprising according to claim 16: 

A step which sends out a demand to said a part of data stream corresponding to said code in said step to 
access to a remote computer. 

A step which receives said portion of said data stream. 

[Claim 19]Including one of http, ftp, or Internet related demand type [ other ], said remote computer is 
connected to the Internet and said demand is the method of said data stream according to claim 18 of 
having access to said portion at least, at least. 

[Claim 20] A way according to claim 1 8 said remote computer accesses a table which makes said code 
correlate with a portion to which said data stream corresponds including said code in which said demand 
was read in relation to said object. 

[Claim 21] A way according to claim 16 said receiving step includes reception of streaming video. 

[Claim 22] A device characterized by comprising the following for accessing video from a note. 

A scanning device constituted so that a glyph code embedded in a video index might be scanned. 

A display mechanism constituted so that a calculation device constituted so that said scanned glyph code 

might be read and said some of videos might be accessed based on this glyph code, and a portion of said 

searched video might be outputted. 

[Claim 23]The device according to claim 22 with which said operation mechanism is constituted 
including network connection by which said calculation device can connect wired ** for any of a 
wireless network being so that said portion of said video can be required via said network connection 
from a remote device. 

[Claim 24]The device according to claim 23 with which said operation mechanism includes a table 
which makes said glyph code correlate with offset of said video. 

[Claun 25]The device according to claim 24 constituted so that said operation mechanism may read said 
table with said scanning device. 

[Claim 26]The device according to claim 22 with which said calculation device includes a palm type 
computer and wireless network connection. 

[Claim 27]The device according to claim 22 with which said scanning device is unified by remote 
control of an auxiliary device. 

[Claim 28]The device according to claim 27 in which said auxiliary device is one of a computer, 
television, and Web TV devices. 



[Translation done.] 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to access to a position with video or other recording media, 
and detection of this position. More specifically, this invention relates to referring to the position in a 
recording medium using the reference marker about the position. As a reference marker, other items 
about a part of note, medium itself, or its position can be used. This invention relates also to use of the 
glyph coding which relates a reference marker with the position to which a recording medium 
corresponds. 
[0002] 

[Description of the Prior Art] Generally a multimedia note preparing system acquires the sound and 
video under meeting, A slide is formed from the acquired data (for example). A live board. Tivoli 
(Tivoli), Moran, T.P., Palen, L.,Harrison, S., Chiu, P., Kimber, D., Minneman, S., vanMelle, W.,which 
is a system performed by (LiveBoard) and. Zellweger, P. . Are alike. "I'll get that off the audio": a case 
of study of salvaging multimedia meeting records. Proceedings of CHI '97 CM,New to depend. Refer to 
York and pp.202 -209. Tivoli is designed support a working meeting rather than a presentation meeting. 
The ink stroke of Tivoli where an indexing is carried out [ sound ] becomes a group note of a meeting 
with the prepared data on the Tivoli slide. The participant who uses laptop "may send" the comment of 
the typed text on a slide of Tivoli. 

[0003] Although the picture and sound of a presentation slide (Classroom 2000) are acquired in the 
classroom 2000 of a similar example, . Video is not used. (Abowd,G.D.,Atkeson, C.G., Brotherton, J., 
Enqvist, T., Gulley, P., and LeMon, J. Investigating the capture, integration.) and access problem of 
ubiquitous computing in an educational setting, Proceedings of the CHI '98 Conference. ACM, New 
York, pp. 440-447;. And Abowd, G.D., Atkeson, C.G., Feinstein, A., Hmelo, C, Kooper, R., Long, S., 
Sawhney, N., and Tani, M., Teaching and learning as multimedia authoring: the classroom 2000 project, 
Proceedings of the ACM Multimedia 796 Confer Refer to ence.ACM, New York, and pp. 187- 198. In 
the classroom 2000, the efforts which a presenter prepares the slide of standard drawing form form are 
called for. A slide is displayed on a live board and note creation is performed using the PDA device with 
which the slide was preloaded. These notes synchronize with the sound and slide with which notes were 
attached by the professor who gave a lecture in front of the live board later. 

[0004]In another example. A forum. (Forum). (Isaacs, E.A., Morris, T., and Rodriguez, T.K. A forum 
for supporting interactive presentations to distributed audiences.) Proceedings of CSCW'94. ACM, New 
York, and pp. 405-416 ~ referring to it ~ it is a system which uses video as a means of the dispersed 
presentation. All the members including a speaker are sitting down in front of the workstation during the 
presentation. A slide must be prepared in a specific form. Notes cannot be attached to a video image 
although notes can also be attached to a slide by the mark drawn with the text and the mouse. 
[0005]In another example. SUTORIMUSU. (STREAMS). (Cruz, G., and Hill, R. Capturing and playing 
multimedia events with STREAMS. Proceedings of the ACMMultimedia '94 ) Conference.ACM, New 
York, and pp. 193-200 - referring to it ~ it is a system for the presentation acquisition which uses the 
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video from an indoor camera. These cameras are used also in order to acquire the arbitrary contents of a 
presentation on a display. This method has the problem that indoor activity displays indistinctly. 
Although notes can be attached to the video stream acquired during the review by adding the comment 
of a text, the note creation under presentation is not supported. Not every system of these can make a 
dialogue unify the live image and presentation data from a camera to a note. 
[0006] Some stand-alone ink and voice note preparing systems are known. For example, FXPAL. 
Dynomite. (Wilcox, L.D., Schilit, B.N., andSawhney, N. Dynomite: A Dynamically Organized Ink and 
Audio Notebook. Proceedings of CHI.) '97. ACM and New York, pp. 1 86-193. Refer to it.; And to voice 
recording paper. The audio notebook (Audio.) to be used Notebook (Stifelman, L.The Audio Notebook: 
Paper and Pen Interaction with Structured Speech. Ph.D. Thesis. MIT, 1997.) referring to it ~ it is 
mentioned. The Philo chat. (Filochat). (Whittaker, S., Hyland, P., and Wiley, M. Filochat: handwritten 
notes provide access to recorded conversations. Proceeding.) s of CHr94. ACM, New York, and pp. 
271-276 " referring to it - a sound is PC computer accompanied by a pen tablet by which an indexing is 
carried out for a handwritten note. .Time in. (NoTime). (Lamming, M., and Newman, W. Activity-based 
information technology in support of personal memory. Technical Report EPC-1991) -103, Rank Xerox, 
EuroPARC, and 1991 were designed double with the sound or video which recorded the user's ink 
stroke. 

[0007]The video annotation system is also known. A marquee. (Marquee). (Weber, K., and Poon, A. 
Marquee: a tool for real-time video logging. Proceedings of CHI '94. ACM, New York, pp. 58-64.) 
Reference is a system of the pen base which attaches notes, looking at videotape. A murky higher 
version, (Minneman, S., Harrison, S., Jassen,B., Kurtenbach, G., Moran,by which correction of 
attaching a time stamp to the digital video stream from a Where WereWe multi-media system is made T., 
Smith, I., and and van. Melle and B. A confederation, of tools for capturing. Refer to and accessing 
collaborative activity.Proceedings of the ACM Multimedia'95 Conference. ACM, New York, and pp. 
523-534. VANA. (Vanna). Harrison, B., Baecker, R.M. Designing video aimotation and analysis 
systems, Graphics Interface '92., [ Morgan-Kaufrnann and ] pp. 157-166. . Refer to it. And EVA 
(MacKay, W.E. EVA: An experimental video annotator for symbolic analysis of video data, SIGCHI 
Bulletin,21 (2), 68-71. 1989. ACM Press ~ referring to it ~ it is a text-based system. Video noter 
(VideoNoter)(Trigg, R. Computer support for transcribing recorded activity, SIGCHI Bulletin, 21(2), 
68-71 . 1989. ACM Press ~ referring to it ~ the sfream from which an activity (video, a number, the 
figure of a white board, a text) differs is displayed, and these are synchronized ~ post-generation is 
required, in order to post a text from a sound or to extract a figure from a white board. These systems are 
limited by the design based on use of the videotape instead of a digital video. No systems of these can 
perform interactive integration to the note of a video image. Sharp Zaums (Sharp Zaums) (Zaums 
Operation Manual. SharpCorporation, 1996) is goods, and is PDA with a digital camera. A digital 
photograph can be taken and it can be made to link to a handwritten note. 
[0008] 

[Problem(s) to be Solved by the Invention] Many systems include a video recovery mechanism. Drawing 
1 illustrates the conventional video recovery device 100 including the video display field 102. The time 
line 105 shows the present position of the video currently displayed, and a user can navigate video by 
the playback 1 15, the rewinding 120, the rapid traverse 125, the top return 130, and the standard VCR 
control containing top delivery 135 grade being provided. However, when detecting the specific position 
in video, and detecting the specific position in video with reference to other items especially relevant to 
a note or video, a problem arises. 
[0009] 

[Means for Solving the Problem]If provided by video supervised during note creation, or a search 
mechanism simple [ a significant part of other data sfreams ], and exact, an artificer will have recognized 
that, as for a note preparing system, increase becomes easy to use the ease of playing. If it says simply, 
this invention provides a multimedia note preparing system which a user can choose a key-frame in the 
midst of a note creation session (a meeting, a presentation, or other activities), or can attach notes. Each 
key- frame and/or notes are acquired and an indexing is carried out to a position of other data streams 
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relevant to video of a note creation session, or note creation. A key-frame and notes which were 
acquired are printed on paper with coded glyph, and a printed sheet, A target key-frame or a glyph code 
relevant to notes is scanned, and it is used in order to access an important portion of video by referring 
to video the indexing was carried out by a target key-frame or notes of, or an important portion of a data 
stream. 

[OOlOJOther purposes are attained by these systems (it is also called a note look) that reach and create a 
note in a digital video and ink. Although this invention is not limited to below, it may be materialized by 
many methods containing any of the following inside they are. 

[001 l]It is an interface for accessing a data stream including an output containing a code relevant to 
each object which specifies a part of data stream corresponding to at least one object corresponding to at 
least a part of data sfream, and an object. 

[0012]At least one object relevant to a data stream is specified. It is the method of determining offset of 
a data stream corresponding to each related object, and forming an index which specifies each offset and 
a corresponding related object which forms an index for access to a data stream. 
[0013]It is a data stream accessing method which accesses a part of data stream corresponding to offset 
which scarmed an object corresponding to a data stream, and determined and determined offset of a data 
stream based on a scanned object. 

[00 14] A scanning device constituted so that a glyph code embedded in a video mdex might be scanned. 
It is a device which reads a scanned glyph code and includes a calculation device constituted so that 
some videos might be accessed based on a glyph code, and a display mechanism constituted so that a 
searched video portion might be outputted and which accesses video from a note. 
[0015]This invention an input of two or more data sfreams to a note creation device, Arbitrary data 
streams which have an acquired portion or a frame in order to refer to it behind, and a user acquired 
(such a data stream) A method of attaching notes related including arbitrary user inputs including a pen 
base input, input video, two or more input videos, a data sfream, a white board input, etc. is also 
included. This method attaches notes, or chooses a frame from one of inputs of a data stream, and 
includes coding and adding glyph which specifies selection or notes which stored, carried out the 
indexing of notes or the selection, and stored it. 

[00 16] A step is repeated to two or more selections and/or notes which were performed in the midst of a 
presentation or data collection. Data glyph expounded on which, chosen and added reproduces a data 
sfream on a point relevant to whether selection which it was printed out by paper, or it was allotted to 
other elecfronic media for selection, and was saved, or reproduction of notes is begun, notes, or glyph 
chosen and related. May use a scanner, in order to scan data glyph, data scanned from glyph is used in 
order to specify and search video, and an added device (or others). Playback of video or other data 
sfreams is started from a position of a data sfream specified by selection and/or notes relevant to scanned 
glyph. 

[0017]Can carry out this invention automatically and a data stream is acquired by this. By specifying a 
key-frame or other data selections, it is simimarized automatically and glyph which pinpoints a position 
of a data stream from which these key-frame or other data selections were extracted, and selection was 
extracted is embedded. Glyph which refers to notes manually attached by user and/or performed 
selection may be formed separately, and may be contained in an output of a selected key-frame. 
[0018]This invention may be carried out on the Internet from which a network, for example, a device, 
acquires download or a broadcasting video sfream for notes or note creation. An output of this invention, 
for example, a paper output which has the glyph chosen and related, is scanned by user. Glyph specifies 
http or other network related addresses to a server, and a server plays video from a position shown by 
scanned glyph, and outputs to PC, a palm device, the Web TV base, or other display mechanisms. 
[00 19] Various embodiments of an output of this invention containing block glyph relevant to a picture 
of notes or a print page are described, A two-dimensional address carpet which has the X-Y position 
which an address carpet of glyph was provided in relation to time line with notes, and was mapped by 
time, Selected access to video access similar to a refrieval interface of arbifrary devices or a summary 
interface of a web browser type product is provided. 
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[0020] 

[Embodiment of the Invention] Many of more perfect worth of this invention and advantages of this 
invention could be easily acquired as he understands this invention more by taking into consideration in 
relation to an accompanying drawing with reference to the following detailed explanation. 
[0021]If the drawing in which the portion to which the same reference number is the same or 
corresponds is shown is referred to again and drawing 2 will specifically be referred to, the computer 
200 of the pen base which performs note creation application 210 is illustrated. Note creation application 
writes in and acquires a picture and includes the main area 215 similar to the page of the note of the 
paper for attaching notes. There is the small video window 220 for seeing active video. Active video can 
be obtained from an active channel, a user changes a channel, and other video streams can be seen or he 
can display a data stream another type on a user or application. Note creation application includes the 
equipment for acquiring the frame shown in the video window, and arranging the acquired frame (for 
example, 225) in the margin of a main area, or acquiring by a set by using a frame as a big background 
image. The "ink" stroke of a user's pen device 230 is also acquired. A time stamp is attached to a 
thumbnail, a background image, and an ink stroke, and it is used in order to provide video or the index 
of other data streams. 

[0022]Active video can be outputted to note creation application via all sauce. Video sources are various 
methods and, for example from an indoor camera or a document camera. It is acquirable from the tap to 
a rear projector, television, the document camera of an overhead slide, VCR, or arbitrary video streams, 
or a data input device arbitrary from the small portable camera with which the pen computer was 
equipped type [ other ]. Many videos or other data sources are outputted to a note preparing system. 
[0023] Drawing 3 and 4 illustrate the screen shot 300 of a note preparing system including the video 
window 320 and the video control 310. There are three buttons for having a dialog with video next to a 
video window. Namely, the top button (channel changer button 322) changes a chaimel. The middle 
button 324 photos a thumbnail (for example, thumbnail 370) in the margin of the note page field 315, 
and the bottom button 326 photos the big picture 390 (or namely, whether it uses for notes, and picture 
possibly expanded in order to see) to the note page field 380. 

[0024]Further the left-hand side of the video window 320 is a set of the VCR type control 360. The time 
line 350 has the pointer 355 in which the present time position of the video currently played is shown. A 
pointer also shows the time position of the data stream (respectively [ for example, input video, a white 
board input, and the pen base input from a user ]) of a large number acquired in the midst of the note 
creation session. The time line also displays indexing information, in order to make browsing and a 
review easy. The exact structures of a note creation device may be arbitrary gestalten, and may use a 
different mechanism from what was explained into this specification, or another control mechanism. 
[0025]By this system, the user can take notes by computer of a pen base, looking at the video which it is 
digitized and is stored in the server (or it is stored locally or stored in other devices, such as a tape and 
CD). A time stamp is attached to a note and it synchronizes with video. The user can photo the still 
picture from video for a note as the thumbnail of a margin, or a ftill-sized picture of a note page. Since a 
time stamp is attached also to these pictures and it synchronizes with video, the duty of an index is also 
achieved. 

[0026]In order to play the recorded video after a note creation session, a note preparing system can be 
used. A playback position is controllable by standard VCR control, for example, a rapid fraverse, or 
rewinding. The specific portion of video can be accessed using the index formed with the pen stroke and 
the still picture of the note. A user chooses a desired index object and pushes a reproduction button (or 
other reproduction mechanisms, such as the double click (Double Click), are used, for example). In 
order to show video recovery time, the time line with notes can also be used (see drawing 1). 
[0027]The above-mentioned note preparing system explains briefly the note look (Notebook) system 
developed by FX Palo Alto Laboratory. In order to acquire a picture, a snap, notes, etc. which apply this 
invention, a note look and other note preparing systems can be used. In order to apply this invention, the 
device of many of other types which support the acquisition, notes and emphasis to video, a sound, 
multimedia, and other data types, or other processings can be used. Only the equipment (it is contained 
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in a device or is accessible) for applying the device and the method of this invention, and/or processing 
which can acquu-e a data stream arbitrary type is required. 

[0028]In order to save bandwidth, it is preferred to use frame rate compression. Automatic change 
detection (for example, slide change detection) may be used, and only the change in a media stream is 
sent to a note creation device. Or the frame rate compression per second of the material in a media 
sfream can also be decreased (that is, the video shot of 30fps may be decreased to 1 or 2fps). Although a 
jitter may appear in the display in the client of a note look by this, The perfect video in a note creation 
device is not indispensable in note creation processing, and it is actually unnecessary in the situation 
where the note maker is attending video (since the note maker can experience what is performed). A 
note creation device may store video/data stream, and video/data stream may be acquired or maintained 
by another server. 

[0029]It is useftil at especially setting out of the conference room which the note preparing system can 
pass an indoor camera and the video from a projector, and wired ** passes a wireless network, and can 
be transmitted to a note look computer. Although it can imagine easily using such a note creation device 
in a conference room, it can seldom imagine owning a device for a user to use it outside of a room 
separately. Therefore, it is important to give a user another means for accessing the video noted down 
and recorded out of the conference room. One of the easiest methods for this is disfributing print-out of a 
note to a user. However, access to the video recorded only now is not given. 

[0030]This invention is embedded at other print-out or other outputs which provide access to the video- 
data stream which used the data glyph of a note, notes, a clip, and a snapshot (a note is called 
collectively) in the midst of the note creation session. Since glyph provides offset of the time of video, it 
can access the specific portion of video. The glyph can code the position of the recorded video relevant 
to these notes. 

[003 l]In order to provide access to video from paper, another application which can use glyph is a video 
summary. One type of a video summary suitable for a prmt, "AUTOMATIC 

VIDEOSUMMARIZATION USING A MEASURE OF SHOT IMPORTANCE AND of the application 
number 09th which applied on March 12, 1999 / No. 276,529. A It is a summary of the comics form by 
Uchihashi of the title FRAME-PACKING METHOD" (all are used into this specification as reference). 
Here, video is analyzed automatically and the key-frame which can be changed in size is chosen as a 
summary. Drawing 4 shows the example of a video summary. A big key-frame is equivalent to the more 
important portion of video. A summary can be arranged to the web page which provides access to video. 
If a key-frame is chosen, video will start playback from the frame. 

[0032]According to a certain embodiment, a summary is printed with the embedded glyph. By this, via 
the computer interface on a web, there is nothing then and access to video from paper is attained. The 
user can scan the glyph embedded into the portion which the video summary printed using the hand 
scanner chose. A scanner is connected to the device or the usual PC like Web TV (Web TV), and these 
access the video / data sfream portion in which the indexing was carried out by the scanned glyph. 
[0033]The user can interface with video by the data glyph embedded at the printed note or video 
summary. Video is accessed by choosing the note or picture acquired from the video which has the 
embedded glyph which refers to the set or other data streams of the position of video, the segment of 
video, and at least one multimedia. 

[0034] D ra w ing 5 illustrates the device of this invention. The acquisition device 400 contains at least one 
input data sfream 410 as which any of a sound, video, a white board, or other data input may be 
sufficient. The user input 420 specifies the portion of the arbifrary data input streams 410, and it is used 
in order to take notes, and/or it attaches the notes relevant to other events in which the input sfream or 
the user took notes. 

[0035]The acquisition device 400 including acquisition/notes mechanism 425 this mechanism. The note 
which the notes or the user whom the user choice portion and user of the input data stream 410 followed 
took is acquired, The time stamp which pinpoints the position in ID, an index, or an input data stream 
for these user choice portions, notes, and/or a note is attached, and these are stored in the stormg device 
430. The acquisition device 400 also contains the channel for sending an input data stream to the storing 
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device 430 for storing. (Or an input data stream may be independently outputted to the storing device 
430, and it may only hold with a server.) 

[0036]The summary mechanism 440 searches the snapshot, the notes, and the note which were acquu-ed, 
and formats these into a form suitable for an output. The summary device 440 can provide independent 
abstract-izing and key-frame selection of the one or more input data streams 410. An output is formatted 
into the web display like the browser device of the personal computer 450, is printed out by the hard 
copy form (paper) 455, or is stored in the electronic format (for example, floppy (registered trademark) 
disk) 460. An output may be outputted to the output device of other types containing other web based 
products, for example, Web TV etc. 

[0037] Dravving 6 illustrates processing of this invention. A note preparing system starts in Step 500. A 
note preparing system is a device arbitrary type which it is commented by the user, including one or 
more of the features shown above, and notes are attached, and receives the selected data stream. The 
user of a note preparing system chooses the data part which attaches notes or is outputted to a note 
preparing system at Step 510. The attached notes or the performed selection is saved at Step 520, an 
indexing is carried out so that selection and/or search of notes can be performed, and at least one 
position of the data stream outputted to a note preparing system is pinpointed. Although not outputted to 
a note preparing system, notes can also be attached with reference to the data stream reproduced in 
relation to a note preparing system. 

[003 8] At Step 530, glyph is embedded with the information which pinpoints the position of either notes 
or selection and one or more specific data streams, and glyph is added to selection or notes (or 
mamtained as reference). At Step 540, notes / selection process, and glyph addition are repeated. 
[0039]Based on the indication of this invention, the variation from which many of flow charts of 
drawing 6 differ may be performed in order to carry out this invention, so that I may be understood by 
the person skilled in the art. For example, before coding each of selection and notes in batch form, it 
may store by performing all of selection and notes by a loop. Processing can be performed parallel or 
serially according to the architecture and programming technique. Important one directs this 
information, including the information as which notes are attached to, selection is carried out, and glyph 
is related with each of notes and/or selection, and specifies a part of data stream corresponding to notes 
and/or selection in glyph. 

[0040] Glyph may also include offset of the information on the position of the related video on a server 
(it corresponds to a specific note set), or the time of the video, and may also contain the pointer of the 
photograph of other types, video, a sound, a document, or data. Since it links to the video currently 
displayed when selection is performed, or when notes are attached, or in order to make glyph link to the 
document m which it is commented whether notes are then attached, using glyph makes sense most. Or 
a user pinpoints the position of video manually (other mechanisms provided by the pop up window or a 
note creation device). 

[0041]If it outputs to the note of paper using a scanning device (for example, handheld computer 
scanner), a user will scan the imaging range of the note of paper and will specify desired selection by 
this. Suitable information is given by the glyph of a scan image and playback of video is started by 
offset of specific time by it. 

[0042]Various embodiments of glyph are used by this invention. According to one embodiment, block 
glyph is used to each object with an index of print-out or other outputs. Each group of the glyph 615 
relevant to the thumbnail (snap) 610 and an ink stroke (the ink strokes 620 and 630 and glyph 625 and 
635) is shown in the example of draxvrng 7_on the page of a note creation device. By processing of 
drawing 6, or processmg equivalent to this, the indexing of the object (a thumbnail and an ink stroke) is 
carried out, and it is added to each glyph. The indexing also of the object of other types may be carried 
out similarly, and the glyph containing the picture containing a background image or data may be added. 

[0043] At a certain embodiment, the position of the video (or other multunedia/data streams) relevant to 
an object is coded by the added glyph on a note page. Or glyph may provide the entry pomt to the table 
which specifies video and offset so that it may explain below. 
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[0044]The ink stroke itself expresses the group of a related ink stroke. Each pen stroke is not that of 
******** with an index at video, and this invention carries out the indexing of the group of a pen 
stroke. Being able to use many methods of carrying out grouping of the pen stroke, these methods are 
within the limits of this invention. For example, the grouping (for example, grouping of a perfect 
sentence or paragraph is carried out using the rule of character recognition and grammar) by syntactic 
structure, the grouping (grouping of the word is carried out) by word recognition, or the grouping by 
time is mentioned. However, since it is preferred that grouping is carried out by time as for a pen stroke, 
its time lag during a group's stroke is small. Therefore, the importance of any losses of indexing 
resolution decreases. It was judged by this invention person that it is the best for a 1 -second interval 
carrying out grouping of the pen stroke. However, this invention is not limited to this. 
[0045]This design is accepted by the arbitrary combination of a video summary or selection, notes, and a 
video summary. In the case of a video summary, block glyph relates to each key-frame of a summary. 
[0046]Drawing 8 is a flow chart of processing of one embodiment of this invention which summarizes 
video and embeds glyph. Video is summarized at Step 700. Video may be the video stream inputted into 
the note creation device, and may be the arbitrary video streams separately recorded in relation to a 
meeting, a meeting, activity, or other events. For example like a voice data stream, the video does not 
need to be the object to which abstract processing was performed, and may summarize and carry out the 
indexing of the product of other types. In the case of products (for example, sound) other than the 
product usually expressed with visual images, other displays which specify an icon, a graph, or a 
product may be included. 

[0047] A key- frame is chosen from the video summarized at Step 710. Arbitrary selection algorithms can 
be used and/or a key-frame can also be chosen with the application of the set of an external reference. 
The indexing of the key-frame is carried out to video or other data streams at Step 720, and it relates 
with the glyph which codes a position with an index. Other data can also be coded to glyph including 
specification of other addresses which can access the data stream of video or a data sfream, http, ftp, or a 
low rank. Or glyph may be coded by the locator number or other identifiers which refer to the front entry 
which can access a low-ranking data stream (that is, Step 720 may also include construction of a 
reference list). 

[0048]Or automatic indexing can also be used in order to determine automatically the important video 
shot (for example, slide change of a white board data stream) which was acquired automatically (using 
for example, a change detection standard), and was formed in the specific note. Such data with 
automatic indexing may be made to link to the time line like the time-line snapshot 860 of draw ing 9 in 
which the slide change 810 and 25:00 shown in a summary corresponds. 

[0049]In Step 730, the indexing also of other notes or selections is carried out, and corresponding glyph 
is provided. In Step 740, an output (for example, paper) is generated, and the video and/or the data 
stream in which the indexing was carried out by glyph from this output are referred to or called. 
[0050]According to other embodiments, the embedded data glyph is provided in the form of a glyph 
address carpet. For example, a glyph address carpet is arranged at the printed note page or the level time 
line under a summary. Drawing 9 illustrates the video summary 800 which has the glyph address carpet 
820 and the related time line 840. In the position to which it is used since the time line 840 maps a 
horizontal position m time, and the glyph address carpet 820 corresponds, linearity or since it is coded 
nonlinear and the indexing is carried out in video, A user cannot be concerned with whether the time has 
an index object or a key-frame (from a note) (from a summary), but can access the arbifrary points of 
video. 

[0051]In order to help navigation as emphasis, arbifrary objects (from a video summary or a note) may 
be displayed on the suitable position on the time line. The various summary objects (key-frame) 860 are 
shown in drawing__9, and it is arranged at the position corresponding to the mdex with which the glyph 
address carpet 820 corresponding to the position of the summary object of video was coded. 
[0052] Since arrangement of a data glyph carpet must be related with the time line so that a user can 
perceive the relation of a key-frame object and the time line, A scan of the object or position on the time 
line will also scan the glyph code corresponding to the object or position on the time line. Therefore, the 
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position of the glyph address carpet 820 may be above the time line, as shown in drawing. 9, it may be 
under the time line, and may be unified with the time line. 

[0053]In order to display linearity or a nonlinear time scale, notes and/or coloring may be performed to 
the time line or a glyph address carpet. In order to access many video (or other data) channels with a 
same or discrete (it is individual or differs) time scale, many parallel address carpet strips can be used. 
This is useful when using a note creation device like the note look which many video channels are 
recorded and is photoed. Many strips can also be used in order to give the selection distributed to the 
long time scale of video. 

[0054]It is chosen and moved by the dispersed selection and the portion of video or other data streams is 
added to other files. For example, in the video time line performed with a glyph data carpet, the portion 
of the time line (and embedded glyph) may be chosen. These selections are dragged, are dropped with 
other applications, and are stored as a separate file, it can be reproduced and arbitrary fiinctions are 
performed. These fiinctions are started by dropping selection with the icon showing the application 
constituted in order to perform the function, or choosing from a pop up window (for example, right-click 
pop up window) fiirther. A fiinction or the application operates by the video clip expressed by selection. 
[0055]Since the horizontal position on an address carpet is mapped at the time relevant to video in the 
case of the address carpet (or other embodiments described into this specification) of the time line, a 
table or other fimctional mapping can be used. This table is stored as a glyph code on paper, and this 
glyph code is scanned and is read by the operation mechanism. This operation mechanism determines 
the index of the video corresponding to the portion of the glyph carpet which referred to and scanned the 
arbitrary portions of the glyph carpet scanned continuously to the read table. Or a table may be stored in 
a server (for example, server which also maintains the video or other data which were digitized). The 
position of video can be provided by another block glyph. 

[0056]Other solution is unifying the label which the video position distributed to the address carpet of 
the time line. For example, the time (a position or an index) of a video name (ID) and video may be 
coded to the time line or other glyph. 

[0057]According to another embodiment, glyph information is embedded at a two-dimensional address 
carpet. Drawing 10 illustrates one embodiment of the two-dimensional address carpet 900. A glyph 
address carpet position is chosen in the accuracy of 1 glyph, and this accuracy is generally about 1/16 
inches on a page, and is equivalent to the resolution of the present electronic computer display. It is new 
or other resolution which uses the art of the improved glyph type may be used. Although the 
embodiment of the beginning of this invention uses a note and a paper interface (print-out) with 
embedding glyph, other embodiments may use all types of output. For example, there is a computer 
display which has sufficient resolution to display the note and glyph coding which are scanned by the 
user. 

[005 8] According to some embodiments, a table is formed and the X-Y position of the two-dimensional 
carpet of the neighborhood of a note object (a stroke, a snap) is mapped at the time of video. Table 1 
shows the illustration table of mapping of the X-Y position of a note object. 



[Table 1] 


X-Y&fi 


x-iS'X h'J-i. I D 




11, 4.5) 




225Z 


(6.5. 4) 




1. 1 


(4.25, 4) 




1. 2 



X-each Y position corresponds to a snap, an ink stroke, or other objects. The selection mechanism which 
accesses a note or the video corresponding to other objects determines the note or object nearest to a 
position of the scanned two-dimensional glyph. 

[0059]Or a table may be more intelligible and the perfect range (field on a two-dunensional carpet) for 
accessing the video position relevant to an object may be demarcated. For example, the video relevant to 
the snap 910 may be accessed from the range of - (2, 0) and - (0, 5.5) (2, 5.5) so that it may be shown by 
the box 915 (0, 0). The video relevant to the ink stroke 920 is accessed by scaiming the glyph in the 
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boundary shown by 925, and the video relevant to the ink stroke 930 may be accessed by scanning the 
glyph of the boundary shown by 935. Bordering size and shape can be calculated by the arbitrary 
methods of dividing the field about the object accessed. 

[0060]The table which related the glyph code to video or the index of other data streams is storable like 
the case of the time line as other structures which are stored in a server with whether it is stored as a 
glyph code on paper, and the digitized video. Or the glyph itself is coded with all the information 
required in order to access video. 

[0061]In other tables, instead of a X-Y position referring a video stream and an index (or offset of 
video), the code embedded at glyph is read by a scanning device, and it is used in order to refer to offset 
or the index of video and video. 

[0062]The position of video can be embedded as an extension address distribution label, or interleave or 
interlace datacode at an address code. For example, the data embedded at glyph, for example, video ID, 
and a position may be coded in other portions of the embedded glyph code. Or may use other methods 
of enciphering or coding the data of a glyph code, and this method, Hecht. To (Hecht). "Parallel 
Propagating Embedded Binary Sequences For Parameterizing Two Dimensionallmage Domain Code 
Patterns in Two to depend Art which was indicated to U.S. Pat. No. 5,825,933 (it uses into this 
specification by reference as a whole) of the title DimensionalSpace" is included. The application of a 
two-dimensional address carpet is applied like the application of a video summary. 
[0063]Drawing 1 1 illustrates one embodiment of the hardware for applying this invention by a network 
envirormient. The calculation device 1000 contains the storing device 1010 which maintains the video 
and/or the data stream which are acquired or played during a note creation session. The server program 
1 020 is constituted so that the portion of the arbitrary data streams acquired according to the demand 
from a browser or the device of other types may be outputted. The calculation device 1000 and the 
server program 1 020 are cormected to the network 1 03 0 (for example, Internet), respectively. A user has 
the circumference scanning mechanism 1060 and the operation mechanism 1050 accompanied by the 
display device 1055. The operation mechanism 1050 is a device of a standard personal computer or the 
Web TV base linked to the network (passing 1040) 1030. In a general embodiment, connection 1040 
with the network 1030 may be made by the telecom company 1035, and any of wireless coimection may 
be [ such coimection ] sufficient as wired **. 

[0064]The output from 1070 of this invention contains various snaps or glyph which note down and 
correspond. The user using the scanning device mechanism 1060 chooses the glyph relevant to specific 
note/snap, and scans the glyph. The operation mechanism 1050 reads the scarmed glyph code, and 
formats a suitable message, and this message is sent to the network 1030 and received by the server 
program 1020. A server program accesses the related part of the data stream stored in the storing device 
1010 corresponding to the scarmed glyph. An access portion is transmitted to the operation mechanism 
1050 via the network 1030, and is displayed on the display device 1055 (a computer screen or standard 
TV). 

[0065]The operation mechanism 1000, the server program 1020, and the storing device 1010 may be 
arranged at the same operation mechanism as the operation mechanism 1000, and may be held at a 
different operation mechanism connected via the network 1030 or other devices. Any of the wireless 
communication system with which wired ** contains a satellite, microwave, and the system of cellular 
or others may be sufficient as connection of a server, an operation mechanism, and the network 1030. 
[0066]The operation mechanism 1050 and the display device 1055, The gestalt of arbitrary electron 
devices including the scanner or other mechanisms for inputting a mobile calculation device and a 
display mechanism, for example, a laptop computer, a palm type computer, or a glyph code may be 
taken. All the places that can use the hole or system contiguous to the dashboard of vehicles, for 
example, a car, delivery, or a repair van or the lobby of a hotel, and a conference room may be equipped 
with the calculation device 1050 and the display 1055. 

[0067] According to a certain embodiment, the scanning device mechanism 1060 is unified by remote 
control of television and/or the Web TV device. Therefore, the user can have a medium arbitrary type 
which has the glyph embedded by the printed summary or the report, the advertisement, or this 
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invention, and accesses the video or other data relevant to a summary, a report, etc. which were printed 
using the scanner. 

[0068]This invention accesses video from the note printed using glyph art, and a summary. This 
invention includes the specific art which codes to glyph a video position and offset of time including a 
table and direct coding. 

[0069]This invention provides access to video from print-out of the note acquired from the elecfronic 
note creation device (for example, note look) using glyph, and contains the following. 

- Relate block glyph with a handwriting stroke and picture of a note page. 

- Form the time line of video using a glyph address carpet, and provide video access from the time line 
of paper by this. 

- Use a two-dimensional address carpet over a note page, and provide the paper interface which operates 
with a scanner like the scanning device of the pen base which accesses video from an index object by 

this. A paper interface is generated by printing an output like drawing 9 on a glyph carpet page. 
[0070]In the case of the printed comics video summary, the time line of video is formed using - glyph 
address carpet, and it provides video access from the paper which is similar to video access from the 
time line of a web by this so that it may illustrate to drawing 4. 

- Use a two-dimensional address carpet over a summary page, and provide a paper interface similar to 
the web interface which accesses video from a key-frame by this. 

[0071]This invention also provides the art of specifying the position of the video on a server, and offset 
of the time of video. Since offset of a video source and tune is coded by these at the same glyph, a scan 
separate although a video source is specified is unnecessary. This invention contains the following. 

- Use of the disfributed label in the glyph carpet address for pinpointing the sauce and the time position 

of video. 

- Use of the interlace code in the address carpet for pinpointing sauce and a time position [0072]The 
experimental result showed that a user liked the combination of the time line and access of an object 
base. In order to access video, using paper has been received, and it turned out that it is useftil also like a 
video summary. The overprint of the note page of a note look has been performed by the glyph carpet 
page by carrying out robust decoding of the address from a handwritten note segment and video frame 
print. 

[0073]The microprocessor programmed by instruction of the conventional wide use, an exclusive digital 
computer, or this indication may be used, and this invention may be easily performed so that clearly 
[ computer technicians ]. 

[0074]Based on instruction of this indication, suitable software coding can be easily prepared by a 
skillful programmer so that clearly [ a software skill person ]. This invention may be carried out by 
preparing an application specific integrated circuit or the interconnection of the suitable network of the 
conventional component circuit so that clearly [ a person skilled in the art ]. 

[0075]This invention contains the computer program product which is the singular number or two or 
more storages which stored the instruction which can be used for programming a computer, in order to 
perform any of processing of this invention they are. Although not limited to below, as a storage A 
floppy disk, an optical disc, A disk arbitrary type [ containing DVD, CD-ROM, a Microdrive, and a 
magneto-optical disc ], A medium or a device arbitrary type suitable for storing of ROM, RAM, 
EPROM, EEPROM, DRAM, VRAM, a flash memory device, magnetism or an optical card, a nano 
(molecule memory IC is included) system or an instruction, and/or data is included. 
[0076]one of the singular number or two or more media which can be computer read ~ it being stored 
and this invention. The software and computer, or microprocessor which controls the hardware or the 
microprocessor of wide use/dedicated purpose computer includes the software which can have a dialog 
with other mechanisms using the result of human being's user or this invention. As such software, 
although not limited to below, a device driver, an operating system, and user application are included. 
Eventually, such a medium that can be computer read includes the software which performs this 
invention as mentioned above. 

[0077]Being contained in programming (software) of wide use / dedicated purpose computer, or a 
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microprocessor, Although it is a software module which teaches this invention and not being limited to 
below, Acquisition of a media stream and notes attachment, generation of the unportant time line of a 
note creation event, The link to the glyph code which displays the link to the point or segment of a 
media stream of a rest frame, and this. It is communication of the result depended on recognition of slide 

change, the abstract of a key-frame, a frame, an ink stroke and creation of the table which makes other 
data correlate with a glyph position, and processing of this invention. 

[0078]It is clear for it to be possible by change of much this inventions and instruction of the above [ a 
variation ]. Therefore, it will be understood that this invention can be carried out by the method except 
having stated to this specification concretely within the claim of this invention. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



TECHNICAL FIELD 

[Field of the Invention] This invention relates to access to a position with video or other recording media, 
and detection of this position. More specifically, this invention relates to referring to the position in a 
recording medium using the reference marker about the position. As a reference marker, other items 
about a part of note, medium itself, or its position can be used. This invention relates also to use of the 
glyph coding which relates a reference marker with the position to which a recording medium 
corresponds. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



TECHNICAL PROBLEM 

[Problem(s) to be Solved by the InventionJMany systems include a video recovery mechanism. Drawing 
Lillustrates the conventional video recovery device 100 including the video display field 102. The time 
line 105 shows the present position of the video currently displayed, and a user can navigate video by 
the playback 115, the rewinding 120, the rapid traverse 125, the top return 130, and the standard VCR 
control containing top delivery 135 grade being provided. However, when detecting the specific position 
in video, and detecting the specific position in video with reference to other items especially relevant to 
a note or video, a problem arises. 



[Translation done.] 
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[Drawing 3] 
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[Drawing 7 ] 



http://www4.ipdl.inpit.gojp/cgi-bin/tran_web_cgi_ejje?atw_u=http%3A%2F%2Fw^^ 11/12/2008 



JP,2002-057981,A [DRAWINGS] 



Page 5 of 6 



Trends; 

[] Free stuff! fBM VtaVoit 



[] MicrosoftTM Research ("/(JsiAfe/ii^s 
Bad Hem as You Might Think") 

[ ] Nobody's Solved the Speech 
Recognition Problem... 
(but can we sell you a recognizer?) 



i 

forge vocabulary 

recogt^i+iijn system 
needs their own 



[Drawing 9] 

























B onnffl 






1^ 




J Drawing l OJ 



Trdnds: ^^jv.,^ 

[li FreeStuff! IBM VIaVolce ~M 
I -i:)Bmo,SR. 

[] MicrosoftTM Reseerch (HfsNof bs^ 
Sad Here ss Vou MIgftf 7fi/n*0 

[] Nobod/B Salved the Speech 
Recognilion PraWem... 
. (biacanvmseHyouaiBcognaaif) 



ailiiiiiiiSiB 



_Tu , c ten mi mS^m If-m !->' 



large vacebiAfoy 
: ;L';,-'!:sr«^!fe,.'thsnS.o«0 = 

Shfiiiiwlpliii |r n J 



[D ra wing 11] 



http://www4jpdl.inpit.gojp/cgi-bin/tran_web_cgi_ejje?atw_u=http%3A%2F%2Fwww4.i... 11/12/2008 




http://ww4.ipdl.inpit.gojp/cgi-biii/tran_web_cgi_ejje?atw_u=http%3A%2F%2Fww4i... 11/12/2008 



